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IB ST E AC T? x V 

♦ This spe'ecK advocates the establishment of empirical 
standards for evaluating" the success* er failure of press |*ipf ormance. 
The ideal me^sii^e, as outlined in the speech, includes information on 
specific news cdver|fe/ the daily operation of the presfe, arid the- 
media institution 9 s past performance. Phis information should' be 
ptecisely gathered; and assembled, and comparisons to global 
Situations should be applied wherever possible. For a complete theory 
of* press criticism, certain independent variables, both from- the 
community in 'which the press institution operates and from within the 
media organization itself , must also be considered^ (HI I) 
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Empirical Measures of Press /Performance 
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iUy-measur6 press perfon qa n ce 



* The beliefs that-rt^ is possible to 

is not, .to 'be sure, universally held., The ptess product is complex^, covering 

" V „ ' , m A 

a multitude of topics and including many types of content. Evaluating, that 

product in terms of community wants and Jfeeds as w^ll as accepted profess-. „ 
;onal standards often seems insuperably difficult. Yet the benefits re- 
sulting fi;om such measurement argue strongly for attempts at overcfLning 



s 



the problems. - # * , < *i 

N The goal of media criticism, in my view, is not merely to identify 
media failings after the' fact or 'serve as a referee for complaints against - 
news, operations. There is considerable value in such activity,, to be 'sure. 
My view is that some' of the media will improve as a result of such criticism.' 
' »" If seems to me,' however, t^hat the real goal of press criticism is— » 

or at least ought to be- 'to understand why the press faj.ls and why it suqceeds. i 

V "t " " 

In other Vrords, press criticism ought/ to lead to a better understanding of 

press behavior. "By evaluating the^ press /we ought to 'come closer to under- 
standing Jhat it is" that make^ one newspaper or one .broadcast news operation . 
* . \ • * i > * 

better, than another. Press criticism ought' to help us understand why, the 

pressfseems to perform better in soie situations or in covering certap kinds 
of stories than in others! • \ t >• 

' , To put it another way^I think ^we need theories which specify how. the 
pf ess "behaves or performs. . Such theeWes ought to specify 'which factors in- 
uenc^press behavior, and the ways iA which such factors interact. Theories \ 
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iof this sort would bje Wased on the bre sent and the Va« . / But they would 

\ * ■ < - ' \ I" . 

help us predict. And* ^hey wbuld allow us to specif^ «foat can be done to 

\ ■ '• I ,. , | " / ' I . , K 

improve the press %n the future. * . / > 

— . T he ories, of fcourlse, are built upon ob*erVat'ioi|» *lThey are general- „ 



izations from tha^ Observation. And they are verified by observation. 
' So a theory of press, performance, quite pimply, requires |observation of 
press performance, j * 8 

^ \ , . i ' . % i 

That doesn'.t mean, ihoweV^r, that measurement- -the classification of 

performance according to |sojni^iterion--is .necessarily -a part of theory 

* " i ' i ♦ " . - \ * 

construction and evaluation ' is possible to have bbseAation without ^ 

I * % \ S 

measurement. In the strijct sense of the tetm, in.other wo|ds,~ it ispos- 

' 1 * § L 

sible tp be eiftpiricajL without employing measurement. M§asi|remejit, or 

i . 1 . * ♦ \ i 1 

classification, of p^rf ounance isn't an essential^rt- of theories of 

* s ' i I - • \ " 

press behavifor. .Only observation is essentials 

1 It is in the ruction of measurement of social phenomena per se 

■ l' * J .' ' Vi- . ' ! * * * \; 

^thal\ rejection of en^iricism in fettfdyirig press behavior finds its most solid 

defense. In other weirds, ithose^who. would argue' that media behavior is immune 

* . v i » 

i to measurement must also rrgue, to bq consistent, that the performance of 
other social institutions" cannot \be njeaisureyl . Rejection • of; measur^ent of 
press performance seejm^ t<j> require iif,my\view, a similar rejeetAojv of 
sociological measurement iln general. ' And Unless some crudial distinction 

. is mad? Wtween social an^-psycholdgical measures, such a* rejection of • ^ 
1 meaisrttement/of pres5 behavior also^ems^td argue fbr rejection of measure- 



mte£t of |pciaj behavior in general. \ 
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Many, of course, take just such a position. Proponents of a qualitative 
methodology, for example, argue that measiitremejit in and of itself distortp soc- 
ial yeality (filstead, 1970). Social researchers using empirical measures 
♦ do not really observe the wgtid-ttbo ut ^h e m rqualitative ^neth odologist^ ar gue. 

They dreate a world in keeping with their measurement techniques. Absolutists 

\ 9 - . 

"argue that no measurement of social pheijom^na isjpossible. J 

While' I'm sympathetic to the criticism of some sociological and % psycho- 
:# • 4 ' * 

logical, measures, I rejeet the notion that measurement is inappropriate. ( I 1 • 

suspect that many of those who fee* uncomfortable Wijth empirical measures of 

» * * * 

press performance reject the* notion as well. % < x 

4 

There is^ of course, ar.long tradition of empirical study of press 

• *** , 

performance under the rubric of bias research; Studies of news play in the 

coverage of "national elections^ for example, were conducted as early as 1952 

(Stempel„ 1975) . They now seem to, be a fairly standard part of election year 

research programs. \, m ^ > 

In organizational research, in sociology and management , where performance 

is defined as the composit goal-directed output of a group, empirical measure- 

, • . t * ■ 

* * 4. 

ment also is the norm (Wofford, Gerloff and Cummins,' 1977). Performance of . 
business ventures, of course, are .regularly assessed in terras of* the financial 
stability they acquirfe through product output, # _ 

Tftefce two research 'examples- -of bias in political news and of organiza- 
tional output— represent two general strategies in measuring the performance 
of complex organizations'. The first strategy is to examine specific organiza- - 
tional output at a ^iven point in time—perhaps after t some crticial change in the 
environment of the organization. .In media research, this ofteii means examining 



coverage of a 
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* 2 * ' 

specific event — such as an election or an element of the election-- 
at a precise paint in tine/ The second strategy is more gldbal. An attempt is 
made in suph analyses, to Assess overall performance of the organizati&i; not 
just performance iji one Specific area at one particular time. 



• Global indices; of course, can fre based on a set of specific measures.' 
Iri newspaper performimke||na^yais, for example, it would be .possible^ to create 
a global measure of a newspaper's performance basW on observations of how that* 
paper covered a variety of specific stories or story topics. A story oti local 
government, one on. urban problems -ancl another sports piece aire only a few of 

the many possibilities. For the most part, however, this strategy has /left b 

» « * * * 
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employed in newspaper performance analysis. • 

■A * * ' 

Perhaps the most ^obvious problem encountered in performance analysis using 

a specif ip meas'ure-- or % global measure based on specific observations- rhas to* 

do with sampling. Any specific story or area chosen is but one of the many 

pieces o^ output of a news organization,. . Qampaign news, in the case of much 

.of the bias research; ;i^crfily ^fraction of the news produced by the -press 
during an election £eri«. *An evaluation of the wljple media institution on the 

. basis of the selected o^put is tenuous attbest, 

Researchers emploj&ng specific measures, of course, are -usually qiiite 
willing to admit that their measure may not adequately rgpect the- overall 
performance of the new" institution. The^nroblem, however, is. that, because 
they have focused only ^n. bne.kind.of output, they ar'e»not able to specify 
what it'** about news' media behavior which produces excellence in one area while 

> not necefsarily in. another. In otheV words/ they can't provide much information, 
which allows for an assessment of • the' link between one kind ofperformance and 
another. ' The questioir'of general performance as a' consequence remains quite a • 



• mystery. 
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A particular problem of specific measures tff preSs performance entailing 

a ju&gmen;t of biaS also is worth noting^ As Rosengren (1977) has noted, bias^ 

measure* per se are lacking in a relevant standard of comparison.. As a result,^ 

!_! ^ * : , „ ! / \- „ . - v y 

researchers must be aware of the possibility that the news event is, in reality, f 

biased. /On any given day in a political campaign, for example, one candidate 

may in reality t\ave done more and said more than another. Balanced news coverage 

of the campaign^ measured" solely in terms of inches devoted to the two candidates, 

would be an inferior' measure? of performance- -unless equality v itself is the -sol|e 

*>vi, : ; , *• ; ! ' ' • 

p - / 

Global measures of performance, of course, are not without their problems. 

■ ThoSe wftch are not based on observations! of performance in specific tasks ruiv> 

the risk of, measuring something quite distinct from actual present performance. 

They may measure instead simple reputation or some othe* aspect of prestige. 

Those WJ^ch are«based on specific observations, as noted, encounter problems 

of representative sampling in making those observations. 

* A- recent example of a global .measure of performance will serve to illustrate 

the first problem. It should be noted, however, that the researchers wjio used this. 

• .. ; v . * : ■ ~ • 

measure— Johnstone, Slawski and Bowman (1976)— did n6t consider it a measure of * 

I 

performance; but rather of prestige or prominence of the news organi&atiorf . There 
•is 4 relationship between this measure of prominence and performance, I" think s 
most will agree, and others might prefer to think of it in performance terms. 

I'm referring, of course., to the n^ional study of newspeople conducted in 
1971 by the National Opinion Research Center under' the direction of the three 
- cited sociologists at the University of Illinois at Chicago Circle. Included 
in the study was a question soliciting from respondents .their nominations for 



^Tlte tKf ee newspapers or other news organizations which they considered to be 

fai rest and' most^reliable . It will surprise few ofyp'u to learn .that the 

* . * • J ' * . 1 

New York Times topped the list with the most nominations. It was followed, 

~ ■ • ' ■ — ™~r ^ 3— L — ; — * 

in order, by the Associated Press > the Washington Post , the Wall Street 
Jourjial and the United' Press International. The list will surprise few of 
you, I suspect, because you .have Other 'evidence these are some of th§ best 
news organizations iji the "country. But they also, fcave Other characteristics. 
They are large. They are eastern. And they are the same five' news oyrganiza- 
£ions which the journalists said they relied on in theijr work. In other words, 
the. Johnstone, et al., question seems to measure performance . But it also 

/ tV . i 

measures ^other things. Unspecifi^d.jy^^ll^fsureS generally have that 

r . v . • . — t 

laracteristic. m f 

measures; 

for media insiitutions can be found in the published product of. the New. England 
Daily Newspaper Survey project (Ghiglione, 1975) .' ,The measures are worth eXBsi- 
ining in some detail because they lack the ^common drawbacks almost unspecified 
global measure*, yet they seem to^ercome the problems' of sampling associated 
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Perhaps the most creative recent attempt at, employing global 



with global indices based ''on specific behavior. \ 

1* 

The measures I'm speaking of ajre included in the evaluative essays evalr. , 

uators wrote about the 100 plus New England dailies. The evaluators didn*t 

m * 

assign numbers to the newspapers or give -them a performance score, though they 

could have. But they did measure the papers in terms of selected criteria. A' 

separate oonteift analysis of these essay? by two of my students and myself has 

* * * 

produced a set of scores for these papers "(Becker, Beam and Russial, 1978). I 

reported to this* division on the reliability of these measures and the use <we 

i 

made of. them in our secondary analyses a year,. ago. * ^ 



/ • • • < • ' • ' • • . 

The New England Survey eyaluators discussed their \task at some length 
before its undertaking. They debated the, Question of relevant criteria of , \ r ~- 
performance and considered various alternatives. The essays they wrote indi-* 
cate they hadn't reachel complete consensus. Jjjit our content^sfrnalysis of tjie -4- 
essays showed each evaluator discussed the following aspects bf^jnress performance 

1. Th6 existence of various kinds of news ih the newspaper.- Specifically, 

j * J, . * 

the evaffuators Rooked for such things as routine coverage of meetings and i t 

of?lcal government, activity, initiative or* investigative pieces, stories pro- 



iding local perspective on state, national or 'international 'events, and^ 
stories of' particular interest to minority groups. The observations werte * 
based on examination of the issues' of the newspap^|for. a six-week perio^. 

2. The thoroughness or depth of these stpries . Some df the copy pro- 

' 9 ' 
vided few details on imfpprtant matters. Other coverage was move substantial. 

- 3. The presentation of the news. Evaluators examined the photography, 
typography a,nd editing of the paper. They examined headlines tb see if they 
matched the, stories under them. They also tried to assess quality^or' writing, 
in general. ^ 

4. The editorial pagprf Evaluators commented on the amount of editoriar 
space devoted to local, state and national* issues, giving particular attention 
to the use of "canned" or rionlocally written editorials. The evaluators also 
locked for balancfc in the use of columnists and other editorial materials. 
* In addition to these four areas, evaluators tried to assess the integrity 
of the news operation. In doing this, they sometimes used data not obtainable 
from the content analysis itself. Fpr example, they talked wherTp^ssible with 
staffers on the papers to learn how frequently copy was influenced by manage-' 
ment. But they also did something 'quite sjjiple. They, looked for puff pieces 



in the papers themselves. This w*(5 concrete evidenc^ the paper^did or did 
not write copy to suit advertisers or other local interests.* 

^ffliai-^particularly striking., jakout the New Engl^ruLiinalysis, it stems 

t. . • * 

to me, is how mych information on press performance the.evalua$tors otfttinfed 

• . \ ' * v ' ' 

directly from a simple v content analysis -of the newspapers. In my opinion, 

■' ' . ' ' " * 

they developed measures of performance which were sensitive to Community 

- > \ 
needs' yet reflective of existing professional' standards* * * . - „ 

, # * \ 
These measures; to be sure, can be improved upon. In the first place, 



the evaluatorsf could easily haVe done more counting. Their measures were* 
Empirical, ta be sure.^They wfere based on obsecration of the press product. 
And they did employ criteria, l^adinj to judgments, judgments, however, * 

were not stilted jln very precise terms. Numbers, are precise. I think it is 
a shame the evaluators didn't rely on them more. They could have counted^ the 
numbejr of lpcall)^ produced editorials and presente<T*heir data on "canned" 
vs. locally-wr.itten editorials as a ratio. They^ could 1 have counted the numbe 
of "puff" pieces. # They" could have counted the number of erroneorus headlines 
or typesetting mistakes. THat would- hay e made the content analysis bf^the 
essays performed by my. two colleagues and I unnecessary. And it would have 

helped us learn even more from the New England project.' • -—- ^ 

/ 

T)ueH4ew Englagd Survey measures of performance were global. They were 
• * t . t 

based on tfye day-to-day operations of the newspapers to'see how the papers 

performed under a variety of situations'. By using a period of this length, 

the pro jectT desig^rs^veided many of the problems of saraf|M.ng » inherent in 

the specific indices discussed earlier. But they also lost, quite simply, 

specificity. . ' % ^ 



v -V 

. . Perhaps the ideal measure of performance would include both\global 

* "| -' ; ( * • • - • V ... 

indices of the sort constructed in the New England project and mor# specific 

measures of the type most often, employed in bias analysis. I^otbe* words , 

1 * ' "" ~~ 



news media crovWage of a specific story or sample of stories could bk ex- 
amined in detail. Information obtained from these specific analyses |ould 
be "combined with the more global infotfeation .to better measure ^rforiSance. 

Perhaps the ideal mfeasure of performance also would include infestation 



on the operation of the news institution in the past to. guard agaihst kkmpfing 

problems. These data could-be obtained from working journalists and others 

, knowledgeable about the area's media, tn other wo«s, reputatiofial measures- 
• • • * 

of performance might be included as well. 

I think community leaders and non- leaders alike haVe important informa T 
tion on press performance. Neither ought to t>e igi#red as potential contribu- 
tors to the ideal' measure of performance.- 

The awards given the media institution unde.r stu<ty--by professional 
groups and community drgani,zations- : alsp tell the evaluator something* about 
the institutions under stud^i - f ' * * 

In constructing our i^eal performance measure, I Thiltk We need to be 
sensitive to thd problem of , a proper standard. There are.^ame^iries objective 
criteria for^inf erring what the press ought to be covering. Funkhouser's 
(1973) analysis of trends 1 in media coverage in the 19$0s is particularly 
suggestive here-. That analysis examined trends in' media coverage of various 

issues during the period and comparecl^hose trends to available statistics > 

.... _ . ., 

on development of the real-woAd phenc^non itself. , Coverage of The Vietnam 
k flar, fpr example, was compared with an|>bjective indicant of the war's 



lUgnituS^^tij^W of troops an Southeast Asian cpuntry—toi see how well the 

J % > T V ' • ■ / 

afera'gte Vefl6tr£& thje war's development. The conclusion Funjchduser reached 

real events were* far' .from, perfectly matched. 

VvJp-- , V' „ •* 

TlTSTldeal Measure of press perfo rmance, t p summarize would 'include » 
• • • > ' • • 1 \ . \* 

information on the way* specific* news events are covered, information on the 

» . ' , /*- ■ 1 ; . 

day -tq- da/ operations of the press, as w.ell £s information on the" media * 
ins ti^t ion's past * performance, -Ae in&itaation would, be precisely gathered 
and assembled.' And it would make comparisons wherever possible to real world 

situations. i . ,\ t *., # " • 

' * • * * 

Vleasuffcs of pr^ss performance are pot sufficient for the, kinds of 

theory testing W'Ve caJJ/sd f or'-hereT Sensitive ifeasuf^s of appropriate' 

* independent variables-^both 'from the community in. which the press institution 

operates and of characteristics pi the media organizatiQn itself ^'-are still % 

* * * 

necessary. It- seems, however, the press s performance measures are the ones - 

most in need of "our attention at present. ^ ^ 

The valine of the kinds/of theories which will' develop' from this- 

empiripal worrit seems >>o me, is that" they can help change the stance 

"of the press critic,. Good "theories allow, .fj??^Borxe.ctive preemptions . 

That, it seems, *to me, is what press criticism is all about-,, •- • 
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